


Usability study in the context of the use of 
European Legislation Identifier (ELI) 


This usability study has been has been undertaken in the frame of the ELI action with the support of ISA?. 

ISA? is a EUR 131 million programme of the European Commission which develops digital solutions that enable 
interoperable cross-border and cross-sector public services for the benefit of public administrations, businesses 
and citizens across the EU (https://ec.europa.eu/isa2/actions/facilitating-exchange-legislation-data-europe_en). 


ISA? supports a large range of actions and solutions. The ISA? solutions can be used free of charge and are 
open source when related to IT. 


a 


Context and objectives 


m This usability study was undertaken on the initiative of the ELI Taskforce and 
coordinated by the Publications Office. For more information on the ELI 


initiative see: http://eurlex.europa.eu/ eli 


m The purpose of this usability study was to gain deeper insight into the needs 
of national legal information system users regarding access to information on 
transposition of EU legislation. 


m The study should identify users, their requirements and expectations. 


m |t should also determine what enhancements, building on ELI 
implementations, could be developed to meet usability interests. 


m This information should provide national legislation publishers with 
information on user needs and possibilities for use of ELI to support these 


needs. 





Phase 1: Research 





Interview & Survey 


E Interviews 
= A maximum of 15 interviews 
1.Stakeholders in the ELI programme 
2.Users 
= Collect insights on: 
e Main goals of implementing ELI on the national websites 
e User profiles 
e Required information 
e Available information now 
e User requirements © ELI 
m A user survey on the national websites 
= 4 questions: 
e Reason to visit the website/ webpage 
e Profile questions 
e Knowledge about ELI 
e Recruiting for future user tests 





a 


User survey questions 


m When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


m When researching European directives, what transposition information would 
interest you? 


m What could this transposition information help you to achieve? Or what extra 
insights might this provide? 


m When looking for transposition information now, do you: 
= usually start from the European directive, wanting to find out if it has been 
transposed by a national law? 
= usually start from a specific national law, wanting to find out if it 
transposes some EU legislation? 


m Profile question 





2. SURVEY RESULTS 





User survey results 


m 1492 respondents 


m Respondent’s language* : 


Not specified: 992 
German: 195 
Italian: 84 
Slovenian: 54 
Romanian: 45 
Portuguese: 33 
French: 25 
Danish: 21 
English: 17 
Bulgarian: 9 
Dutch: 7 
Swedish: 3 
Greek: 2 
Spanish: 2 
Finnish: 2 


Respondent’s language (if specified) 


* Automated language detection by Google Translate 


"a 
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Interest in transposition information 


ARE YOU INTERESTED IN FINDING THIS TRANSPOSITION 
INFORMATION ON THIS WEBSITE? 


m No, that kind of information is not useful forme m Yes, that would be helpful 








"a 


User Profile 
HOW ARE YOU VISITING THIS WEBSITE TODAY? 


m As a citizen, private individual m As a professional = As an academic 








User Profile - Focus on occupation 


WHAT IS YOUR PROFESSION? 
FOR PEOPLE WHO IDENTIFIED AS ‘PROFESSIONALS’ 


mLawyer, legal professional 
= Civil servant, policymaker (non-EU) 
m Employee in the private sector or self-employed 
= Other 
s Member of the judiciary 
Legal enforcement officer 
m Law librarian 
= Legal publisher 
J ournalist, media professional 
m European Commission official, employee of an EU 


institution 
= Translator 





Not specified 





User Profile - Focus on occupation 


WHAT IS YOUR OCCUPATION? 
FOR PEOPLE WHO IDENTIFIED AS ‘AN ACADEMIC’ 


mw Student 


E Researcher 


E Professor/teacher 





E Not specified 





m 


How users approach the transposition relationship 


WHEN LOOKING FOR TRANSPOSITION INFORMATION 
NOW, HOW DO YOU USUALLY START? 


= From a specific national law to find out if it transposes EU legislation 
= From the EU directive to find out if it has been transposed by a national law 








m 


How users approach the transposition relationship? 


When looking up information on a national law, what information 
relating to this transposition would you like to find on our website? 
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mFrom a specific national law to find out if it transposes EU legislation 
mFrom the EU directive to find out if it has been transposed by a national law 





B 


When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


2. A LINK TO THE EU DIRECTIVE (ON EUR-LEX) THAT IS 
BEING TRANSPOSED BY THIS NATIONAL LAW 





Academic Citizen Professional 


m From the EU directive to find out if it has been transposed by a national law 
8 1% m From a specific national law to find out if it transposes EU legislation 


overall 
score 





When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


1. AMENTION OF THE EU DIRECTIVE THAT IS BEING 
TRANSPOSED BY THIS NATIONAL LAW 





Academic Citizen Professional 






80% mFrom the EU directive to find out if it has been transposed by a national law 
0) 


overall 
score 


m From a specific national law to find out if it transposes EU legislation 





m 


When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


3. DOES THIS NATIONAL LAW 
TRANSPOSE EU LEGISLATION? 





Academic Citizen Professional 


mFrom the EU directive to find out if it has been transposed by a national law 
7 5% mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





m 


When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


4. LINKS TO OTHER NATIONAL LAWS THAT WERE 
TRANSPOSED FROM THIS SAME EU DIRECTIVE 





Academic Citizen Professional 


o m From the EU directive to find out if it has been transposed by a national law 
65 Yo mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





m 


When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


5. DETAILED TRANSPOSITION INFORMATION: HOW DO THE SECTIONS 
OF THE NATIONAL LEGISLATION RELATE TO THE ARTICLES OF THE 
DIRECTIVE(S) 





Academic Citizen Professional 


0 mFrom the EU directive to find out if it has been transposed by a national law 
54 Yo mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





m 


When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


5. MORE SPECIFIC INFORMATION: THE DATE OF EFFECT, 
THE DATE OF TRANSPOSITION OR THE AUTHOR OF THE EU 
DIRECTIVE 





Academic Citizen Professional 


mFrom the EU directive to find out if it has been transposed by a national law 
48% mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





m 


When looking up information on a national law, what information relating to 
this transposition would you like to find on our website? 


5. ASUMMARY OR THE FULL TEXT OF THE EU DIRECTIVE 
THAT IS BEING TRANSPOSED BY THIS NATIONAL LAW 





Academic Citizen Professional 


0 mFrom the EU directive to find out if it has been transposed by a national law 
44 Yo mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





How transposition information is used 


Faster overview of the legal position in 
other EU Member States as well; helps to 
clarify the content of the directive.' 


From the EU directive to find out if it has been transposed by a national law 





How transposition information is used 


Easy to draw comparisons between EU 
directives and national transposition. 
Better interpretation of national 
transposition. Identifying the purpose of a 
standard. ' 


From the EU directive to find out if it has been transposed by a national law 





How transposition information is used 


Understanding about the transposition of 
EU law could improve and make it easier to 
comprehend certain decisions. ' 


From the EU directive to find out if it has been transposed by a national law 





How transposition information is used 


More efficient research, certainty about 
the comprehensiveness of research, 
increased transparency.' 


From the EU directive to find out if it has been transposed by a national law 





How transposition information is used 


I need the information to see which legal 
instruments created by the European Union 
have seemingly positive and negative 
influences on our day-to-day lives and 
other EU citizens. ' 


From the EU directive to find out if it has been transposed by a national law 





How transposition information is used 


‘Faster access to information, answers to 
questions. ' 


From a specific national law to find out if it transposes EU legislation 


How transposition information is used 


For the purposes of preparing national 
legislation.’ 


From a specific national law to find out if it transposes EU legislation 





How transposition information is used 


‘Increased clarity when interpreting 
national law.’ 


From a specific national law to find out if it transposes EU legislation 


How transposition information is used 


Verifying how a specific national provision 
based on an EU directive has been 
transposed in a different Member State 
(more strictly? Differently?).' 


From a specific national law to find out if it transposes EU legislation 





How transposition information is used 


Helps with standardising terminology 
when translating legislation in accordance 
with national law. Thereby makes it 
possible to align EU legal texts for the 
purpose of translating national acts of 
transposition. 


From a specific national law to find out if it transposes EU legislation 





How transposition information is used 


Better tracking of legal context, better 
understanding of framework legislation. ' 


From a specific national law to find out if it transposes EU legislation 





m 


How users view legislation? 


When researching EU directives, what transposition information would 
interest you? 


100 





mFrom a specific national law to find out if it transposes EU legislation 
mFrom the EU directive to find out if it has been transposed by a national law 





m 


When researching EU directives, what transposition 
information would interest you? 


1. IS THIS DIRECTIVE TRANSPOSED BY NATIONAL LAW - YES 
OR NO? IF SO, BY WHICH NATIONAL LAWS? 





Academic Citizen Professional 


0 mFrom the EU directive to find out if it has been transposed by a national law 
90 Yo mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





m 


When researching EU directives, what transposition 
information would interest you? 


2. IF THE DIRECTIVE HAS NOT BEEN TRANSPOSED INA 
CERTAIN MEMBER STATE, WHY NOT? 





Academic Citizen Professional 


6 1% mFrom the EU directive to find out if it has been transposed by a national law 
0 mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





"a 


When researching EU directives, what transposition 
information would interest you? 


3. ALINK TO THE TEXT OF THE NATIONAL LEGISLATION 
FROM DIFFERENT MEMBER STATES THAT TRANSPOSES THIS 
DIRECTIVE 


a 49% 62% 





Academic Citizen Professional 






60% mFrom the EU directive to find out if it has been transposed by a national law 
(0) 


overall 
score 


m From a specific national law to find out if it transposes EU legislation 





"a 


When researching EU directives, what transposition 
information would interest you? 


4. THE TEXT OF THE NATIONAL LEGISLATION (IN FULL, IN 
PART OR IN SUMMARY) DIRECTLY ACCESSIBLE ON THE 
WEBSITE 


58% 





Academic Citizen Professional 


0 mFrom the EU directive to find out if it has been transposed by a national law 
58 Yo mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





"a 


When researching EU directives, what transposition 
information would interest you? 


5. DETAILED TRANSPOSITION INFORMATION: HOW DO THE 
ARTICLES OF THE DIRECTIVE RELATE TO THE SECTIONS OF 
THE NATIONAL LEGISLATION 


55% 


50% 





Academic Citizen Professional 


mFrom the EU directive to find out if it has been transposed by a national law 
5 1% mFrom a specific national law to find out if it transposes EU legislation 


overall 
score 





Phase 2: PERSONAS 





Persona 1 


> Quick stats 


= Occupation: Member of judiciary 
= EUR-Lex use: 3 times a week 
= User level: Advanced 
= Country: Slovenia 
> About 


This person works at the Supreme Court of 
Slovenia. It is really important for her to 
understand the European regulations. 
That is why she does comparative studies 
of the transpositions of other countries that 
have a similar legal system to Slovenia, 
mainly Germany and Austria. 


| like to start my 
research on a European 
level, because it’s easy to 
find all 
the transposed legislation 
from there. 





> Goals 


1, 


2; 


E 


Both the directive itself and the transposed 
legislation are important. 


Understand European rules 


> Actions 


Searches national implementations using: 


1. 


‘National Implementing Measures’ on 
EUR-Lex; 


information stored in the directive itself; 


the title of the transposed act to search in 
Google or on the national legal information 


systems. 


> Expectation 


1. 
2; 


A multilingual interface would be great. 


Detailed information on transposition would 
be useful. 





Persona 2 


> Quick stats 


Occupation: Governmental 


EUR-Lex use: Daily 


= User level: Advanced 
= Country: Portugal 
> About 


This person has a law degree and works for 
the customs office. 

Because she’s dealing with border issues 
every day, it is really important for her to 
have easy access to other countries’ 
legislation. 


Sometimes it’s easier 
to just google than to use 
EUR-Lex or Curia 





> Goals 


m 


To understand a European directive 


2. Tosee how a directive has been implemented 
in different countries and how that law is 
being treated. 

> Actions 

1. First uses Google. With the results from 
Google she can look for the right content on 
EUR-Lex or on Curia. 

2. Checking national laws one by one to find out 


from which EU directive they have been 
transposed. Because she can’t find that 
information in a centralised way. 


> Expectation 


1. 


To have access to each national law in a 
common language to compare 


To get access to transpositions 
from one central point 





Persona 3 





> Quick stats 


= Occupation: Legal Deputy Director in 
private sector 


= EUR-Lex use: Often 


= User level: Advanced 
= Country: Italia 
> About 


= This person is a legal director in a 
pharmaceutical company and releases a legal 
Opinion in the matter of public procurement 
and public tenders. 


Italian law has jurisprudence 
contradicting 
national legislation, because 
transposition was 
badly done. 


> Goals 


1, 





E 


Primary goal: to find if, what, when and mainly 
how a directive was transposed into national 


law 
2. Compare Italian legislation with other 
transpositions and with the original directive 
to check the legitimacy of the Italian legislation 
> Actions 


Finds the reference of the transposed directive in 
the Italian legislation and uses this to search the 
directive using Google. 


> Expectation 


i; 


To be able - in one click - to see if the directive 
has been transposed in other countries. 


Have a direct link to the national 
websites or to EUR-Lex. 








"= 





Persona 4 


> Quick stats > Goals 


= Occupation: Law librarian 1. Find how a particular directive has been 


= EUR-Lex use: Couple of times per month transposed into Irish law 


2. Ifa directive hasn’t been transposed into 


= User level: Advanced 
Irish law yet, check how it has been 
" County: Ireland transposed in other countries, e.g. UK 
> Actions 
> About 


1. Uses WestLaw UK (a subscription 


= This person is a librarian for a law firm in database) to search European law 


Ireland. Her main duty is to maintain the 


firms’ library collection. 2. Looks for directives on a European level 


She also assists the legal staff with research starting from national legislation 
queries and request and notifies them 


when new legislation has been passed. > Expectation 


1. A link to all transpositions and the date of 


If we cannot find the transposition. 
Irish transposition, we'll look 2. To see how other Member States - 
in the UK to see how it has particularly the UK — have transposed the 


been transposed there. directive. 





Persona 5 





> Quick stats 
= Occupation: Academic 


= EUR-Lex use: Depends on the project but 
not on a daily basis 


= User level: Advanced 
= Country: Austria 
> About 


= This person is a legal assistant who works in the 
field of European Law. He is also working as a 
research assistant. 
Both of his jobs require that he checks other 
countries’ legislation and whether the 
European directive has been transposed or 
implemented correctly. 


What | cannot do is to see 
which part of the Austrian law 
is transposed from which 
part of the directive. 





"= 


> Goals 
1. Primary goal: to know if a certain directive has 
been implemented by countries 
2. Compare the original directive with the 
national transposition 
> Actions 
1. Start from the national database when he 
looks for Austrian law or French law 
2. But for all the other legislation and countries, 


he will definitively start from EUR-Lex 


> Expectation 


1. 


It would be really helpful to have a direct link 
from the European directive to the national 
database. 


It would be good to have detailed 
transposition information. 





Phase 3: User testing 





User testing 


m Description 
= Test of the full version prototype: https:/ / invis. io/ EHITLMCYA 
= Test of the quick win prototype: https:/ / invis.io/ DR9LL75F3 
= Test of the widget: https: / / invis. io/ 3F9T17Z5U 


m Objectives 
= Make sure the user can identify specific elements on the 
webpages: 
e The most recent version of this law 
e The version of this law that was in force in 2012 
e Any other national laws that transpose the same EU directive 
e Reference of the law in any other legislation 





Test full version prototype 





National Justice Portal 


DOCUMENT INFORMATION 


CHANGES IN LEGISLATION 


BROWSE RELATED DOC 


LINK TO EU DIRECTIVE 


FULL TEXT: Order No. 1179 of 21.09.2016 


Order on working conditions for crew members in 
service on aircraft and for their employers 
Pursuant to § 40g, paragraph. 1 and 2 and § 149 paragraph. 10 of the Law on Aviation, see. Act no. 


1036 of 28 August 2013 as amended by Act no. 602 of 4 June 2016 fixed by order pursuant to § 152 
paragraph. 1 


Art. 1 
Field of application 
5 1. This Order applies to crews serving on board Danish, civil aircraft and for their employers. 
PCS. 2. An aircraft is considered Danish when it is Danish registered or operated under a Danish 
operating permit. 
§ 2. The provisions of this Order shall apply only in connection with the crew member's 


performance of work on board the aircraft and only if the work covered by the crew member 
certificate rights or be considered part of the employment relationship. 


Art.2 
Definitions 
§ 3, For purposes of this Order 


1) Crew member: a person under the law on aviation or regulations made under that holds a 
certificate issued or approved by Transport and Construction 


2) Flight Crew: A certified crew entrusted with tasks of importance for the lead of an aircraft in 
flight duty time. 


3) Cabin crew member: A certified crew assigned tasks other than transfer of an aircraft in flight 
duty time. 


4) Fmninver / worker A holder nf a certificate has the right ra immeriiately exercise the rioht 


TABLE OF CONTENTS ~ 


m 


1. Table of 
contents 


2. Changes in 
legislation 


3. Document 
information 


1.1 What do you on the right side of 
the page? 


1.2 How would you use this if you're 
looking for a specific piece of 
information? 


2.1 If you want to know if this page is 
showing you the most recent version 
of this order, where do you look for 
this? 


2.2 Where would you find the version 
of this order that was in effect in 
January 2016? 


3.1 Can you tell me from which day on 
this order was applicable? 


3.2 Under the title ‘Identification’ you 
will also find the European legislation 
identifier or ‘ELI’. Do you know what 
this is? 





m 


Test full version prototype - more questions 


4. Linkto EU | 4.1 Do you think this order is an 
directive implementation of a directive from 
the European Union? 


4.2 What do you think would happen if 
you clicked the button that says 
Transposition measure’? 


4.3 What would happen if you clicked 
the ‘EUR-Lex’ button? 


5. Browse 5.1 Where would you find a list of 
related doc decisions that were made based on 
this order? 


5.2 Under ‘Regulations which 
implement EU Directive’, click the 
first item. What do you see now? 


6. 6.1 If your national justice portal was 
Appreciation | like this, would that better help you in 
your work? 





Test quick win prototype 


DOCUMENT INFORMATION 


Status (Current / History) 


Identification 


European legislation identifier: 


Year of issue: 


Date of signature: 


CHANGES IN LEGISLATION 


Subsequent amendments to 
Regulation 


Changes in / repeal 


Legislative regulation concerns 


LINKS TO EU DIRECTIVE 


3.3.3373 332 


BROWSE RELATED DOC 


Regulations, which implement 
EU Directive 


Table of contents 


> 
n 
u 


Order on working conditions for crew members in service on 
aircraft and for their employers 


Pursuant to § 40g, paragraph, 1 and 2 and § 149 paragraph, 10 of the Law on Aviation, see. Act no. 1036 of 28 August 
2013 as amended by Act no. 602 of 4 June 2016 fixed by order pursuant to § 152 paragraph. 1: 


Art.1 


Field of application 


5 1. This Order applies to crews serving on board Danish, civil aircraft and for their employers. 
PCS. 2. An aircraft is considered Danish when It Is Danish registered or operated under a Danish operating 
permit 


52. The provisions of this Order shall apply only in connection with the crew members performance of work on 
board the aircraft and only if the work covered by the crew member certificate rights or be considered part of 
the employment relationship. 


Art. 2 
Definitions 
5 3. For purposes of this Order: 


1) Crew member: a person under the law on aviation or regulations made under that holds a certificate issued 
or approved by Transport and Construction 


2) Flight Crew: A certified crew entrusted with tasks of importance for the lead of an aircraft in flight duty time. 
3) Cabin crew member: A certified crew assigned tasks other than transfer of an aircraft in flight duty time. 

4) Employee / worker: A holder of a certificate has the right to immediately exercise the right certificate in a 
particular company, generally considered employed by the company during the period that the certificate right 
must be exercised in the company concerned. If there are exceptional circumstances, Transport and 


Construction Authority on application decide that the starting point may be waived. 


5) Employer: Any natural or legal person who has an employment relationship with the crew member, and has 
responsibility for the company. 


6) mSv, millisievert, which is a measure of the average cumulative dose of background radiation, exclusive 


Radon, as a person in relation to its weight receiver. A Sievert is one joule per. kg, and a millisievert corresponds 
to one joule per. ton. 


Art. 3 
Generality 


5 4, Safety and health work in the company handled through cooperation between the employer, any supervisors 
and employees crew members 


PCS. 2. The employer must provide employees crew all necessary information affecting health and safety, and to give 


1. Table of 
contents 


2. Changes in 


legislation 


3. Document 


information 


4. Link to EU 


directive 


5. Browse 
related doc 


6 


Appreciation 


1.1 Do you think this is better or 
worse than having it next to the text? 
Why? 


2.1 How can you know that this is the 
most recent version of this order? 


3.1 Can you tell me which ministry 
was responsible for this piece of 
legislation? 


4.1 Do you think this order is an 
implementation of a directive from 
the European Union? 


4.2 Click on the first directive. It takes 
you to another website. Do you prefer 
to view the European directive in the 
same website or doesn’t it bother you 
that it opens in another website? 


4.3 There are two links for each 
directive; is it good to have both links? 
Which one do you prefer? 


5.1 Under ‘browse related doc’ 
‘Regulations which implement EU 
directive’ click the first item. What 
happens? 


6.1 Which of the two concepts would 
you prefer? 
| | 







Test widget prototype 


En EI 
eli.legilux.lu 


ELI 


Législation 


> Contenu 


BEE Version publiée 
Règlement grand-ducal du 9 juin 2006 





sumer hii mème fonction de travailleur down 


Nous Henri, Grand-Duc de Luxembourg, Duc de Nassau, 

Vu la loi du 17 juin 1994 concernant la sécurité et la santé des travailleurs au travail, telle que modifiée par les lois du 6 mars 1998 et du 13 janvier 2002 modif 
concernant la sécurité et la santé des travailleurs au travail; 
- Vu les avis de la Chambre de Commerce, de la Chambre des Métiers, de la Chambre de Travail, de la Chambre d'Agriculture et de la Chambre des employés pri 


Notre Conseil d'Etat entendu; 
De l'assentiment de la Conférence des Présidents de la Chambre des Députés; 
Sur rapport de Notre Ministre du Travail et de l'Emploi et de Notre Ministre de la Santé et après délibération du Gouvernement en Ca 


Arrétons: 
Chapitre I®", — Dispositions introductives 


Ani", 
Aux fins du présent règlement grand-ducal on entend par: 


L emiistren: ie ministre ayant le travail dans ves atrbutions; 


2. aloi modifie du 17 juin 1994s: la lot du 1? juin 1994 concernant ia sdcurme et la santé des travail au travail, telle que modifiée par les lois du 6 mars 1988 et du 13 janvier 2002 modifiant la oi du 17 4 
santé des travailleurs au travai, 


3. etravaiieurss tous les salarié tls que déni à l'article 1de la loi modifiée du 24 mai 1589 sur te contrat de travail ainsi que les stages les appren et les dèves et etudiants occupés pendant les vat 
4 eposte ò mauer: tout poste de travail rempiasant les conditions de article 17-1 de ia loi mas du 17 juin 1994 concernant les services de santé au travail 
$ ie Commission corsultgtwes: Commission constative d apprécaton reiste aux sanctions det formations dispensées aus furs travailleurs désignés, commision tede que nie à Farce 3 du presen 
6. ele Comté consuftaif: Comitè conautat relati a formation des travailieurs dés, comté tei que dé à l'article 10 du présent règiement grand Jucat 





Chapitre Il. — Catégories d'entreprises; 
Définition du nombre suffisant de travailleurs désignés; 
Entreprises dans lesauelles l'emoloveur peut assumer lui-même la fonction de travailleur désiené. 


DOCUMENT INFORMATION 


Status (Current / History) 


identification 


European legislation identifier: 


Order: 
Accession: 
vor: 
Type: 
Ranking: © 


Dates 

Year of issue’ 

Date of signature: 
Publication Date: 
Date of effect: 

Date of transposition: 


Classifications 
Index: 
EUROVOC descriptor: 


Subject matter: 
Directory code: 


fety 


CHANGES IN LEGISLATION 
BROWSE RELATED DOC 


LINK TO EU DIRECTIVE 


1. Changes in 
legislation 


2. 
Appreciation 


1.1 Where could you find links to 
previous versions of this piece of 
legislation? 


2.1 As you can see, this concept 
contains the same information as the 
second prototype. Do you think there 
are advantages to either the second or 
the third prototype? 





Detailed results 





2.1 Reading results 





Fail 


Succeed very slowly in a 
roundabout way 


Succeed a little slowly 


Succeed quickly 


Not applicable 


Needed help all the time 


Needed help a lot 


Needed some help 


Needed no help 


Not applicable 


"a 


ae ee 


Pulling hair out 


A lot of frustration 


Some frustration 


No frustration 


Not applicable 


8/4/2017 
51 





m 


2.2 Usability test participant 1 


m Quick stats > General appreciation 
" nnaen Lega AMISSI = | want to see the interaction between 
= Website use: Daily iawe 
= Country: Slovenia 
> Podium 
aa 1. Protot 3 - Widget 
= For my work I’m interested in laws that eile 
concern investment in public health and in 2. Prototype 1 - Full version 
construction safety. 3. Prototype 2 - Quick win 





"a 


2.2.1 Usability test participant 1 - prototype 1 - full version 


El ind Hi Bl Li | 





| see the table of contents 


) | | will use it to access an article directly 


Find the information in the ‘document information' tab: status 


2.2 [Go directly to 'changes in legislation' 

3.1 Find the information in the 'document information' 

3.2 He think it’s an identification number 

4.1 Find the information but it could be more clear 

42 “N/A 

4.3 N/A 

5.1 Go directly to ‘browse related doc’ 

5.2 | will find various national laws that implement this EU directive 

6.1 E m quite happy with the current website. | miss some information in the 


_ prototype like other national laws based on the EU directive 





2.2.2 Usability test participant 1 - prototype 2 - quick win 


RESET SE 
1.1 1 It’s better to have it on the top 
2.1 FIT | can see it thanks to the status 








3.1 Could not find the 'see more' button 

4.1 | Can tell it’s an implementation of an EU directive 
4.2 | prefer to have it load in the same page 

4.3 | Don't see that there are two different links 

5.1 | don’t mind that it opens a new window 


6.1 a a The first concept is more appealing 





2.2.3 Usability test participant 1- prototype 3 - widget 


= Ch ia 


Eine directly the previous version of the law 


= are advantages to the 3 prototype because it will be easier for the 
people to adjust quickly 









a 


2.3 Usability test participant 2 


m Quick stats > General appreciation 
= Occupation: Project management = The first thing | want to see is the date 
= Website use: Daily and if it’s the last version 
= Country: Romania = | would like to be able to compare 
changes in legislation 
= | would like to see practical cases and 
m About decisions based on this piece of legislation 
= |'m an environmental specialist. | am also = | would like to see what has changed in 
finalising law school. I’m interested in legal the law with all the extra info (like the 
issues regarding environmental matters date) 
particularly about protected areas. | work = | would like to have a print and save in pdf 
with different national websites. | also make button 
considerable use of the European Court of = | would like to see how a directive has 
J ustice website and idrept.ro been transposed in other countries 
> Podium 


1. Prototype 1 - Full version 
2. Prototype 3 - Widget 
3. Prototype 2 - Quick win 





2.3.1 Usability test participant 2 - prototype 1 - full version 


Time 


| see the table of contents 











N/A 
First goes to 'document information’ then in ‘changes in legislation’ 
Go directly to ‘changes in legislation’. But would like to see each version 


next to each other to easily compare them. 


Did not find the information. She will go to the end of the document. 
But it’s just a matter of time to get use to the new navigation. 


Did not know what ELI is 


ad və 2 > Fl. the 
N m N m NF 


4.1 Did not find the information. It should be more clear 
4.2 Yes and no because in Romania all law is transposed 
4.3 Expect to be redirected to EUR-Lex 

5.1 Go directly to ‘browse related doc’ 

5.2 | will find subsequence of the EU directive 


It would be helpful. But it will be more credible if it’s on the EU 
website rather than on the Romanian website. 


= 
m 





2.3.2 Usability test participant 2 - prototype 2 - quick win 


It’s better to have table of contents fixed to access it all the time. But | 
like the fact that the font size is bigger. 


| can see it thanks to the status. But it could be clearer 
Find directly the ‘see more’ button 


Can tell it’s an implementation of an EU directive 


| prefer that it opens a new window 


The first concept is better because it’s not too loaded with information. 
You can close or open information 








2.3.3 Usability test participant 2 - prototype 3 - widget 
Eare e 
1.1 | 4 Find directly the previous version of the law | 


2.1 N/A 









2.4 Usability test participant 3 


m Quick stats > General appreciation 

= Occupation: Editor = NA 

= Website use: Daily 

= Country: Slovenia > Podium 

1. Prototype 3 - Widget 

an 2. Protot 1 - Full i 

= |’m editor of online portals about the public a a goad mcrae 

sector and accounting for different 3. Prototype 2 - Quick win 


professional target groups 





"a 










I see the table of contents 





“E 
| | law. 


Find the information in the ‘changes in legislation’. I'm confident it’s the 
£ last version because the date is 2016 
a Go directly to ‘changes in legislation’. 


a“ not find the information. He will go to the end of the document. 


2.4.1 Usability test participant 3 - prototype 1 - full 
| will use the find function' to find the find what I’m looking for in the 























| Maybe an identification number. But he’s not sure. 











| ft could be but he’s not sure 


| 4.2 i | This article has been transposed by this directive 


| Expect to be redirected to EUR-Lex 

















Go directly to ‘browse related doc' 








| This directive wastransposed into 55 other national laws 


6.1 a Yes it will be helpful. | like this one because it has the two levels: 











| national law and EU. 














m N 
3 = 
5 > 


4.3 


6.1 





NO 
Cc 


sability test participant 3 - prototype 2 - quick 


The previous was better because you don’t have to scroll back to get to 
the table of contents 


= 
® 


| cannot see at a first glance 
Find directly the information and the 'see more' button 


| think it is transposed because it looks similar to the EUR-Lex website 


| like to have two different windows. | can simultaneously switch from 
one window to another. It’s more functional but less aesthetic 


| don’t need both links. Link to EUR-Lex will be enough. 


| prefer that it opens a new window 


The first concept is more appealing. It has something new 








E 


2.4.3 Usability test participant 3 - prototype 3 - widget 






Find directly the previous version of the law 


The 3" prototype is more user friendly because you can hide it if you 
don’t need it 





m 


2.5 Usability test participant 4 


m Quick stats > General appreciation 
= Occupation: Legal advisor = | would like to have the details of the 
= Website use: Almost on a daily basis transposition 
= Country: Denmark 
> Podium 
m About 1. Prototype 2 - Quick win 


= |’m working for a consultancy company on | 
chemical legislation and product safety. 2. Prototype 3 - Widget 
For my job I have to find out if we have 3. Prototype 1 - Full version 
national legislation on specific matters and if 
other countries have the same legislation 





"a 








2.5.1 Usability test participant 4 - prototype 1 - full 





| see the index and the order 





| will look for the link and try to go to the part of the text I’m looking for 








Find the information in the ‘document information' tab: status > 
applicable. But | would prefer the label 'current'to ‘applicable’ 





’ a 


Go directly to ‘changes in legislation’. 














F have often looked for that feature but | don’t understand the label 
‘transposition measure’ 





Expect to be redirected to EUR-Lex 





Go directly to ‘browse related doc'. But I’m not sure of the label 
decisions’ 





| really like this page. It shows all the documents which implement the 
| directive or a part of it. 











Yes it certainly would 








2.5.2 Usability test participant 4 - prototype 2 - quick 


LEST EU EEE 







| prefer at the top because it’s what I’m used to. But as long as it’s there 
_ it’s OK for me. 





Er Find it with the status. | like the fact that | don’t have to click on it to 


_ see the information 


| 3.1 2 tt Find directly the information and the 'see more' button | 


| 4. 1 ‘ Can tell it’ san implementation of an EU directive | 


N/A 











| don’t need both link. Link to EUR-Lex will be en 


| prefer that it opens a new window. Because | can switch to a different 
eee iene without messing the original document. | 














2.5.3 Usability test participant 4 - prototype 3 - widget 






| Find directly the previous version of the law 


It's very good to get the info in my own language or in English 





2.6 Usability test participant 5 


m Quick stats 


= Occupation: Librarian 

= Website use: Daily 

= Country: Luxembourg 
m About 


= | work for a law firm as a librarian. 
It is quite essential for my job to find up-to- 
date laws for Luxembourg and the EU. 


"a 


> General appreciation 


It would be nice to highlight as well the 
part of the directive which is 
implemented 


It would be very nice to have access to 
related case-law like in EUR-Lex 


> Podium 
1. Prototype 1 - Full version 
2. Prototype 3 - Widget 
3. Prototype 2 - Quick win 





a 


2.6.1 Usability test participant 4 - prototype 1 - full 


| 
| 


= It will help me navigate in the document 








F see the the table of contents 








| I would look at the top to see if it’s a consolidated version 





| Go directly to ‘changes in legislation’. It could be nice to have the period 
_ of effectiveness (but it doesn’t have to) 





| Did not find the information 





| have no clue 








| Did not find it right away 





| Did not understand the label ‘transposition measure’ 








Expect to be redirected to EUR-Lex 





Go directly to ‘browse related doc’. 





| I'ma little bit confused by the label under the date 








| It will definitively be helpful particularly with the highlight and related 
| document 








2.6.2 Usability test participant 4 - prototype 2 - quick 


1.1 
2.1 
3.1 






| don’t care 
It doesn’t tell me it is obsolete so it should be the latest version 


Find directly the information and the 'see more’ button 


4.1 Can tell it’s an implementation of an EU directive. Because there are links 
to the EU directive. 

4.2 N/A 

4.3 It’s dangerous to have both because the text might not be the last 
version. 

5.1 | prefer that it opens in the same window 

6.1 | like the first one better because it’s all linked to each other and you 


navigate in the same page 








E 


2.6.3 Usability test participant 4 - prototype 3 - widget 


COR Sa 
1.1 DE po Find directly the previous version of the law 
2.1 | like it because | could easily find the information on other national 
| websites 








a 


2.7 Usability test participant 5 


m Quick stats > General appreciation 
= Occupation: Legal research 


= It will be even more helpful to have the 
= Website use: Daily 


. reference of which part of the EU 
= Country: Austria directive has been transposed 


= Providing just the provision would be 


m About even better 
= l'm a legal researcher at the university. 
l'm comparing different settings and > Podium 
approaches to EU law. 
| monitor how EU Law has been implemented 1. Prototype 2 - Quick win 


2. Prototype 3 - Widget 


3. Prototype 1 - Full version 





2.7.1 Usability test participant 5 - prototype 1 - full 


SS 2 at 
1.1 N/A 





1.2 N/A 

2.1 Si Find the information in the ‘document information’ 

2.2 a Go directly to ‘changes in legislation’ 

3.1 DR Did not find the information 

3.2 BE | can find it in EUR-Lex. It’s a classification. 

4.1 il Yes. Go directly to ‘link to EU directive' 

4.2 D | Think the label is curious. | would expect to see how the directive has 
been transposed. 

4.3 a Expect to be redirected to EUR-Lex 

5.1 ia Go directly to ‘browse related doc’. 

5.2 a | see national measure that implement this directive 

6.1 i It will be helpful. That will be an improvement. 









At first glance it’s better. But | don’t mind having it on the left. What’s 
weird is having the information on the left. 














NU Yes because it’s applicable 
| Find directly the information and the 'see more' button 
ae Can tell it’s an implementation of an EU directive. 


N/A 





























The link to EUR-Lex should be enough 











| don’t mind 



































| like the second concept better because | don’t have to open the menu 























2.1.3 Usability test participant 5 - prototype 3 - widget 






Find directly the previous version of the law 


It depends on who your target group is. It would definitively be helpful to 
have a similar tool from one national website to another. 





a 


2.8 Usability test participant 6 


m Quick stats > General appreciation 


Upon: Lawygr = | would like to have access to an 


" Website use: It depends on the case I'm English translation of the national law 
working on 
= Country: Italy > Podium 
1. Prototype 1 - Full version 
m About | 
= |'m a tax litigator lawyer in Italy. 2. Prototype 2 - Quick win 
| sometimes look for measures of 3. Prototype 3 - Widget 


implementation, sometimes | just want to 
compare implementations. 





2.8.1 Usability test participant 6 - prototype 1 - full 


CRT BE a 


1.2 


2.1 


2.2 


3.1 


3.2 


4.1 


4.2 


4.3 


5.1 


5.2 


6.1 


| see the the summary of the law 
| usually read the whole document but it will help me navigate in it 


Go directly to ‘changes in legislation’. Yes I’m sure it’s the latest 
version. 


Go directly to ‘changes in legislation’. 

Find it directly in ‘document information’ 

Don’t know what it is 

| think so because there is a transposition date 

| think it’s the initial piece of the legislation at the EU level 
Expect to be redirected to EUR-Lex 

Go directly to ‘browse related doc’ 

Those are the rules and provisions related to the main order 


For me it would be more useful to see it for other countries 





2.8.2 Usability test participant 6 - prototype 2 - quick 


EI BEE ET ea 






| like this one better because it’s more organised 





Yes because there is no subsequent amendment. But it was clearer in the 
first prototype. 


EE Find directly the information and the ‘see more’ button 
“4. 1 1 | Can tell it’s an implementation of an EU directive. | 
“4. 2 ee | don’t care. But | miss the comparison tool. | 


One link will be enough. | don’ t care if it’s the text or the link to EUR- 
Lex. 




















‘don’t mind 





| like the first one better because the layout is clearer and it gives extra 
info 





2.8.3 ee test participant 6 - prototype 3 - widget 






Find directly the previous version of the law 


| don't like it because it doesn’t provide the translation 





m 


2.9 Usability test participant 7 


m Quick stats > General appreciation 
| STAR nen = | would like to be able to click on Art. 2 
= Website use: Daily and see a history of what happens for 
= Country: Ireland this article. 
= What would be nice would be to be 
m About able to see all the changes that have 
= |'m collecting information to manage internal been applied to each article 
database. 


| keep an eye on all legal developments in 3 
Ireland and the EU. And | summarise the = „Podium 

legislation and analyse the impact of 1. Prototype 2 - Quick win 
developments on various groups. 

| maintain the know-how system of my 
department. 3. Prototype 3 - Widget 


2. Prototype 1 - Full version 





"a 


2.9.1 Usability test participant 7 - prototype 1 - full 


RRQ EC RE RE 









| see the the table of contents 


| It depends whether I know which article I’m looking for. 


Yes definitely, the more information the better 


Go to changes i in legislation’, That iS helpful. 


| will use our one database 


Don’t go to document information. Look in the text to find this 


information. Don’t understand the menu on the left. Think the label 
‘document information’ is too generic. 





| don’ t need the ELI number for my work. Dates are more > important. 


| Yes because you have links to EU directive 


| don’t understand what ‘transposition measure’ is. But it will be very 


helpful to have this information. 


| Expect to go to the official version of the EU directive 


Go directly to ‘browse related doc’. 


| assumed that is how the directive was transposed into other law 








2.9.2 Usability test participant 7 - prototype 2 - quick 


EDLER EEE 






| like this one better 


|| Yes because of what you see in changes in legislation 


Go to look at the bottom of the text. And | will need to make some 
research here. 


“It's an implementation of an EU directive, because there is a transposition | 

date. 
4 2 + De | would prefer it to open a new page 
4.3 POTT EUR-Lex will be more helpful 
[51 | | NWA | 


| | | | 
6.1 ie i | like this one better because all the information is available 



































E 


2.9.3 Usability test participant 7 - prototype 3 - widget 


MEEN 
1.1 N/A 
2.1 I. 1. | like the second approach better 





a 


2.10 Usability test participant 8 


m Quick stats > General appreciation 
| weciparlan: ae 6 = It could be nice to have headline next 
" Website use: Daily to article to find quickly what you're 
= Country: Austria looking for. 
= The most important feature will be 
m About related document and changes in 
= I'm working as a researcher in the food legislation and the consolidated text. 
industry. | make formulations of new 
products and I’m in charge of the legal "= It’s nice to have the document 
information that appears on the packaging. information visible at first sight. 
So | have to know what the limit for the > Podium 


different ingredients are . 


| use different national websites and Eur-Lex. 1. Prototype 3 - Widget 


2. Prototype 1 - Full version 


3. Prototype 2 - Quick win 





2.10.1 Usability test participant 8 - prototype 1 - full 


Eeg e deso eee 














| see the table of contents of the different pieces of the legislation 





It depends if | know which part I’m looking for. But if I’m looking for 
information in Art. 25 then it would be quite handy, but it will need a 
headline. 


m 
N 








> 
e 





Go directly to ‘document information’. Yes I’m sure it’s the last version 
because the status is applicable 








Go directly to ‘changes in legislation’ 


N 
N 








w 
He 


Find it directly in 'document information' 








It’s the number of the directive | 


w 
N 



































| 

| 4.1 | | think so because you have a European number (ELI) 7 
42 I’m not sure J 
| 4.3 BE | Expect to be redirected to EU directive in EUR-Lex 7 
| 5.1 BE Go directly to ‘browse related doc’. | 
| 5.2 E ‘It looks like of other legislation based on the directive N 
| 6.1 u | Yes it look quiet simple and direct. And it’s really user friendly. Most of 

| the time the Austrian transposition isa translation of the EU directive. | 











2.10.2 Usability test participant 8- prototype 2 - quick 





| prefer to have it on the right to keep it visible when I scroll 
Yes because it as the status applicable 

First scroll down then find the “see more btn” 

N/A 

N/A 


One link will be enough and | will prefer to have the EUR-Lex link, 
because then | can have the information in different language and | 
can download pdf. 


| would prefer to have it in the same interface. It would be ok to be 
redirected to another website if I’m familiar with the website where 
I’m redirected. 


| would like to have a combination of those two 








a 


2.10.3 Usability test participant 8 - prototype 3 - 
DRS 

LI nu Click directly on ‘changes in legislation' 

2.1 ee *: nice to be able to minimise it 








Result summary 





Result summary 
m There is not an obvious winner that stands out 


= 15 position with 11 points*: Prototype 3 - widget 
= 2nd position with 9 points*: Prototype 1 - full version 
= 3 position with 7 points*: Prototype 2 - quick win 





* When prototype is in 1% place it gets 2 points, in 2" place it gets 1 point and in 3 place 0 points. 





Recommendation 





4.1 General recommendation 


m The widget (prototype 3) could be a good option as: 

= it doesn’t interfere with the actual Member State legislative 
portal; 

= it will be easily implemented by the Member States; 

= it keeps consistency between the different Member State 
portals and therefore makes it easier for the user to look for 
the information; 

= it could be developed by the Commission and handed over to 
Member States which will just add the widget to their existing 
websites. 





4,2 Modification to improve the prototype 


m In the three prototypes: 
= Make it more visible that it’s the latest (applicable) version; 
= Make it more clear that it’s a transposition from a EU directive; 
= Provide additional information about ELI; 
= Only keep the link to EUR-Lex in ‘links to EU directive’. 


m In prototype 1 - Full version: 
= Change label of ‘transposed measure’; 


= Change the radio button in button in ‘changes in legislation’ and 
‘browse related doc’; 


= Add a button for subsequent amendments and make it disabled if 
there are none; 


= Make the document information visible at first sight; 


= Create a tour widget to quickly show the functionalities of the 
prototype. 


m In prototype 2 - Quick win: 
= Make the ‘see more button' more visible 





4,3 Other features request 


m Multi-lingual widget 
@ Law translation at least in English 
m History for each article 
m Advanced comparison tool: 
= Compare changes for a specific article 
= Compare one article with the related EU directive article 
= Compare a national law article to another country law article 


m Have access to practical cases and decisions based on this piece 


of legislation 





Final wireframes 





5. Final wireframes 


m Prototype 1 - Full version: https: / / invis.i0/ MDAI7KN2N 
m Prototype 2 - Quick wins: https: / / invis.io/ UPAI7KX3V 


E Prototype 3 - Widget: https: / / invis.io/ SPAI7L7MF 





